The minimal clinically important difference of app‐based electronic patient‐reported outcomes for hay fever

Abstract Background Hay fever is a common allergic disease, with an estimated worldwide prevalence of 14.4% and a variety of symptoms. This study assessed the minimal clinically important difference (MCID) of nasal symptom score (NSS), non‐nasal symptom score (NNSS), and total symptoms score (TSS) for app‐based hay‐fever monitoring. Methods MCIDs were calculated based on the data from a previous large‐scale, crowdsourced, cross‐sectional study using AllerSearch, an in‐house smartphone application. MCIDs were determined with anchor‐based and distribution‐based methods. The face scale score of the Japanese Allergic Conjunctival Disease Standard Quality of Life Questionnaire Domain III and the daily stress level due to hay fever were used as anchors for determining MCIDs. The MCID estimates were summarized as MCID ranges. Results A total of 7590 participants were included in the analysis (mean age: 35.3 years, 57.1% women). The anchor‐based method produced a range of MCID values (median, interquartile range) for NSS (2.0, 1.5–2.1), NNSS (1.0, 0.9–1.2), and TSS (2.9, 2.4–3.3). The distribution‐based method produced two MCIDs (based on half a standard deviation, based on a standard error of measurement) for NSS (2.0, 1.8), NNSS (1.3, 1.2), and TSS (3.0, 2.3). The final suggested MCID ranges for NSS, NNSS, and TSS were 1.8–2.1, 1.2–1.3, and 2.4–3.3, respectively. Conclusions MCID ranges for app‐based hay‐fever symptom assessment were obtained from the data collected through a smartphone application, AllerSearch. These estimates may be useful for monitoring the subjective symptoms of Japanese patients with hay fever through mobile platforms.


| INTRODUCTION
Hay fever is one of the most common allergic diseases affecting approximately 30 million people in Japan alone. 1 The global prevalence of hay fever is estimated at 14.4%, and this number is expected to increase in the future. 2,3 Hay fever presents with a wide variety of symptoms including allergic rhinitis, conjunctivitis, and dermatitis. 4 It is also a highly multifactorial disease, influenced by three major categories of factors: (1) environmental, such as pollen count and particulate matter; (2) lifestyle-related, such as diet, tobacco use, and exercise; and (3) host-related, such as age and biological sex. [5][6][7][8] Currently, the main treatment for hay fever revolves around symptom suppression and progression through the long-term use of antihistamines, steroids, and eye drops. 4,9 However, due to the variability in its pathological underpinnings, severity, and symptomologies, every patient responds differently to prescribed treatment.
To provide an effective regimen, providers must closely monitor patients' response to treatment and preventive measures, as well as the aforementioned factors to tailor treatment choices. 10 Assessment of interventional responses mainly comprises evaluation of patients' subjective symptoms and clinician's objective findings. 11 For a highly variable disease such as hay fever, an approach to comprehensively quantify patients' subjective symptoms using patient-reported outcomes (PROs) has been frequently utilized. 12,13 PROs allow providers to quantify one's health status related to a disease of interest through questionnaires with information on symptomology, treatment, and overall health provided directly by patients. 14 For hay fever, commonly used PROs include the nasal symptom score (NSS), non-nasal symptom score (NNSS), and their sum, referred to as the total symptom score (TSS). 15 Additionally, several disease-specific PROs related to patients' quality of life (QoL), including the Rhinoconjunctivitis Quality of Life Questionnaire (RQLQ) and the Japanese Allergic Conjunctival Disease QOL Questionnaire (JACQLQ), may further aid in the assessment of a patient's hay-fever status. 13,16,17 A holistic view on PROs may be particularly useful as a quantifiable metric that reflects a variety of hay-fever manifestations, including ocular and nasal symptoms, as well as their resultant effects on QoL. 18,19 Longitudinal monitoring of PROs to evaluate changes may allow clinicians to objectively measure treatment efficacy for hay fever in a personalized manner. 13,20 In clinical trials, a novel treatment is generally accepted as more effective when the analysis of its targeted effect shows statistical significance compared to an existing alternative or control. 13 However, statistical significance often does not equate to clinical significance, particularly for subjective clinical markers including PROs. 13 In bridging the gap between statistical significance and clinical significance, researchers deploy the concept of minimal clinically important difference (MCID). 21,22 MCID represents the minimal improvement gap that can be regularly perceived as a benefit of value to patients, and an increase in PRO above the designated MCID would represent a clinically meaningful treatment effect. 21,23 The designated MCID values can vary between patient populations due to the differences in demographics, disease severity, and patient expectation. 24,25 Therefore, the determination of appropriate MCID values requires the selection of MCIDs calculated from data from a patient population as close as possible to the target patient population. 24 Our previous studies have elucidated numerous aggravating factors and stratified the symptoms of hay fever using a mobile health (mHealth) smartphone app for hay-fever research-AllerSearchreleased in February 2018. 4,12,20,26 AllerSearch is equipped with a function to assess users' hay-fever symptoms through collecting ePROs (electronic PROs), including the NSS and NNSS. 4,12,20,26 mHealth, to date, has shown great promise in monitoring daily symptoms and treatment effectiveness with minimal intrusion. [27][28][29][30][31][32][33] Utilizing mHealth and a PRO with a widely agreed-upon MCID value, it may be possible to accurately track meaningful changes in patients' symptoms and provide tailored treatments that are effective for individuals. 4 However, robust MCID values for hay-fever outcomes collected through AllerSearch are yet to be determined.
In this study, appropriate MCID values were determined for NSS, NNSS, and TSS for evaluation of the subjective symptoms of hay fever based on data collected through a mHealth smartphone app, AllerSearch.

| AllerSearch smartphone application
The self-developed smartphone application, AllerSearch, was initially released as an iOS-based application on ResearchKit in February 2018, and the Android version was released on August 26, 2020, under a consignment contract with Juntendo University Graduate School of Medicine and InnoJin, Inc., Tokyo, Japan. 4,20,34 The Aller-Search is freely available on the Apple App Store and on Google Play.

| Study design
MCIDs were calculated using both anchor-and distribution-based approaches using data from a previous large-scale, crowdsourced, cross-sectional observational study conducted between February 1, 2018, and May 1, 2020 using AllerSearch. 4 This study was approved by the Independent Ethics Committee of Juntendo University  Table 1 and 2), JACQLQ, 17 and work productivity. This study used data on answers to the questionnaires related to symptoms of hay fever that were entered for the first time in the app.
The NSS consists of five items pertaining to rhinorrhea, nasal congestion, nasal itching, sneezing, and interference with daily life. 26 The NNSS consists of four items pertaining to itchy eyes, watery eyes, eye redness, and itchy ears and mouth. 26 Each item of the NSS and NNSS is scored on a 4-point scale as follows: 0 = no symptoms, 1 = mild symptoms, 2 = moderate symptoms, and 3 = severe symptoms. 15 The TSS (0-27) was derived from the sum of NSS (0-15) and NNSS (0-12). The face scale score was scored on a 5-point scale depicting emotions ranging from 0 (fine) to 4 (crying). 11,17 The stress level was scored on an 11-point visual analog scale from 0 (none) to 10 (most stressful).

| Inclusion and exclusion criteria
Participants who resided in Japan with reported hay fever were included in this study. Participants without hay fever, with an unknown diagnosis, with missing questionnaire data related to symptoms of hay fever, face scale, and stress level were excluded from this study.

| Anchor-based analysis
Anchor-based methods are used an external indicator as an anchor to assign patients to clinically relevant categories. 23,36 An estimation of MCID for specific PRO measures is recommended to be based on multiple anchors. 37 Here, the face-scale score of JACQLQ Domain III and today's stress level owing to hay fever were used as anchors in determining MCIDs of NSS, NNSS, and TSS.
The face scale score was used as a 5-scale anchor to categorize participants into five severity categories. 11 There is no established method for determining meaningful change on an 11-point stress level scale. In a prior study, a 2-point change was used as a meaningful difference on an 11-point scale. 23,38,39 Based on this method, the level of stress was assessed according to a 5-scale anchor and the participants were categorized into five severity groups. 39 To assess anchor eligibility for determining MCID, Spearman's correlation coefficient and the number of participants in each severity category were calculated. 23,39 An anchor was considered suitable when the correlation coefficient between the anchor and total of NSS, NNSS, and TSS was ≥0.3 and the sample size within a severity category was at least 10 entries. 23,39 Mean differences between adjacent categories of the anchor provided estimates of the MCID. The anchor-based MCIDs were estimated using medians with interquartile ranges (IQR). 23,39

| Distribution-based analysis
Distribution-based methods were presented as supporting data to facilitate the interpretation of the anchor-based results. The following two distribution-based methods were used as an additional confirmation of the MCID. 37

| Standard error of measurement
The formula used for calculating the standard error of measurement (SEM) was "standard deviation (SD)*√(1 − r)", where r was the recommended PRO test-retest reliability. 23,24 In this study, intraclass correlation coefficient (ICC) was used as a metric for test-retest reliability to determine SEM for calculation of MCID. 24 ICCs were calculated by comparing baseline and subsequent measurements of NSS, NNSS, and TSS from patients who indicated no change in the face scale score and stress level. 40 A PRO score difference smaller than the SEM is likely to represent an error of measurement rather than a real change. 23,37 In cases where the median anchor-based MCID was less than SEM, the SEM was used as the MCID. 23

| Half a standard deviation
Half a standard deviation (SD) at baseline was assumed to be equal to the MCID. 41,42 The 0.5 SDs into each category were calculated. The calculated 0.5-SD was used to determine MCID ranges. 43 NAGINO ET AL.

| Statistical analysis
Ranges of MCID were recommended rather than single MCID estimate values. 23 As such, the results of the anchor-based MCID estimates calculated using medians with IQR and distribution-based estimates based on 0.5-SD and SEM were summarized as MCID ranges. 23 To compare the characteristics of the participants in each group, continuous variables were presented as means, and SD and categorical variables were presented with proportions. The STATA software package (v. 17.0, Stata Corp, College Station, TX, USA) was used for all the analyses.

| Participants' characteristics
In total, 7590 participants were analyzed in Tables 1 and 2 Both the face score and stress level scale showed that the more severe the category, the higher the proportion of younger and female participants. Participants in the more severe group were more likely to state their medical history as unknown.

| Estimation of the MCID with the anchor-based method
The Spearman's correlation coefficients between anchor groups and NSS, NNSS, and TSS were higher than 0.  Table 5.

| Estimation of the MCID with two distributionbased methods
The results of the distribution-based analysis are shown in Table 5.  (Table 6). Based on the ICCs, the SEM was found to be 1.8 for NSS, 1.2 for NNSS, and 2.3 for TSS (Table 5).

| Estimation of the MCID ranges
The estimated SEM of 1.8 for NSS was larger than the 25th percentile of median anchor-based estimates of 1.5, and the SEM was  22 Compared to previous studies, the per-item MCID for the collected app-based TSS was larger (0.27-0.37 points per item). 11 However, study designs that target patients visiting medical facilities may experience a higher degree of sampling homogeneity than the average population, and distribution-based MCID ranges, which are highly influenced by the variability of collected PROs, may be narrower in such research designs. 45 This study utilized a highly inclusive sampling method with no exclusion criteria based on hay-fever severity, age, and location owing to the advantages of app-based recruitment. Of the population providing data for this study, 83.6% (6344/7590) reported hay-fever symptoms. Therefore, the resultant distribution-based MCID is thought to better reflect the physiological undulations of hay-fever symptoms experienced by the population, which consists primarily of symptomatic patients, and these MCID ranges may be more appropriate in assessing hay-fever PROs for the general public than previously reported values. It is crucial to utilize MCID values that best reflect the targeting population, and hence, we suggested an T A B L E 1 Demographics and characteristics of participants by face scale score category.   MCID range rather than a single value for each PRO. 39 In future studies that require assessment of meaningful changes to clinical status in hay-fever patients, the suggested MCID range allows clinicians to select an MCID value suitable for the setting (for example, the lowest MCID value for specialty clinics) and accurately monitor hay-fever progression.
In this study, two methods were utilized to calculate MCID: (1) distribution-based method and (2)   Abbreviations: N/A, not applicable; SD, standard deviation.
two approaches, results from the anchor-based method, which utilizes an external clinical marker termed the "anchor", are prioritized in current practices. 22,24 This is in part due to the unwanted influence of various characteristics of the subject pool when relying solely on the distribution-based method. However, anchor-based methods are prone to subjectivity from researchers during "anchor" selection, 22 and the distribution-based approach is relatively bias-free and free from the subjectivity of researchers. 22,24 Therefore, combining the two strategies to determine MCID values is preferred. 22,24 For an anchorbased method, it is recommended to assign multiple anchors for outcomes, and for a distribution-based method, calculating MCID using SD and SEM may yield a more accurate MCID that reflects the constant variability of PROs. 24 Additionally, a larger sample size may help estimate a more accurate MCID range. 25 In this study, we selected two anchors-face score of JACQLQ domain III and stress score-for the anchor-based method and utilized both the SD and SEM for the distribution-based method, approaching MCID calculation in a multifaceted manner. The smartphone app-based, large-scale design is also a strength of this study and the resultant MCID ranges. 4  in unreached patients who are not receiving appropriate diagnosis and treatment for hay fever. 32 The combination of MCID and mHealth apps has the potential to provide appropriate hay-fever treatment with remote subjective symptom assessment to previously unreached patients. Fifth, hay-fever severity may vary depending on when the information was registered, but this study also includes data from various other seasons. However, most of included hay-fever symptom data were registered in February and March. Therefore, the MCID in this study may be applicable to cedar and cypress pollen-allergic patients, which constitute the majority of the Japanese hay-fever patients.
In conclusion, our study results suggest MCID ranges of 1.